In digital imaging, the dynamic range of a complementary metal-oxide-semiconductor (CMOS) sensor may, at times, be insufficient to accurately represent outdoor scenes. This may be especially true in the more compact sensors which may be used in mobile devices, such as in the camera on a mobile telephone. For example, a typical sensor used in a mobile device camera may have a dynamic range of approximately 60-70 dB. However, a typical natural outdoor scene can easily cover a contrast range of 100 dB between light areas and shadows. Because this dynamic range is greater than the dynamic range of a typical sensor used in a mobile device, detail may be lost in images captured by mobile devices.
Previous attempts to extend the dynamic range of image sensors each have their own advantages and their own disadvantages. For example, some previous approaches involve combining two frames, one with a shorter exposure time and one with a longer exposure time. In this approach, the frame with the longer exposure time may be used to show details in dark regions while the frame with the shorter exposure time may be used to show details in bright regions. This approach may yield good results for objects which are stationary, and may produce excellent wide dynamic range (WDR) pictures. Unfortunately, when moving objects are photographed, the time difference between the two different frames used to create such an image may cause motion ghost effect. This approach may also require a full frame memory in such a device.
To reduce such motion artifact, another approach is to start the integration of the short exposure frame immediately after readout of the long exposure frame. To explain how this scheme works, let us assume the short exposure time is S line-time and the long exposure time is L line-time. With a typical rolling shutter operation, an imager will start integration of first row at time 0 and then second row at one line-time later, and so on. At time L line-time, we will readout the first row for the long-exposure frame and immediately start integration for the second, short exposure frame, followed by readout and reset of second row one line-time later, and so one. Thus, for example, at time L+S line-time, a readout of the short exposure frame of row 1 will occur, while at time L+S+1, a readout of row 2 of the short exposure frame will occur. This continues until both frames are read out completely. While motion artifact may be reduced compared to the previous approach, this approach will still require S lines of memory buffer in order to reconstruct the complete WDR frame.
In order to minimize motion artifacts from dual-frame approaches and to reduce amount of required line buffers, a different scheme may be used where different integration times are implemented within the same readout frame. For example, one possible scheme is to have two rows having a longer integration time, Tlong, and another two rows having a shorter integration time, Tshort. This scheme may be used for color sensing imagers where the color filter sub-pattern usually comes in pairs, such as the well-known Bayer pattern, which has a 2×2 sub-pattern with two green pixels in one diagonal and a red and a blue pixel on the other diagonal, as illustrated in FIG. 1.
Similar to other approaches, the rows with Tlong are used to show details in dark regions while rows with Tshort are used to show details in bright regions. With such approach, motion artifacts are minimized and no line buffer is needed. However, this approach will result in resolution loss in the vertical direction. Another improved approach is presented in U.S. Pat. No. 8,059,174 where the long and short integration time are tightly interleaved, i.e., some pixels in a row will have longer integration time while the other pixels in the same row will have shorter integration time, resulting in a much better resolution in the vertical direction. However, this still results in some resolution loss, in both vertical and horizontal directions. In a similar scheme, one row of pixels with long integration time may be used and the next row of pixels may have short integration time, with both pixels having the same color filter. However, this approach results in a different aspect ratio and vertical resolution loss.
Yet another approach to this problem may be to use non-linear response pixels, such as logarithmic pixels. While such pixels provide very high native dynamic range, images created using these pixels suffer from other issues, such as higher fixed-pattern noise (FPN).
Besides different integration time, other techniques such as in-pixel manipulations can also be implemented to achieve WDR. For example, in the so-called lateral over-flow integrated capacitor (LOFIC) scheme, a small capacitor is added in each pixel to accumulate charges when a particular pixel is near saturation. In this scheme, higher dynamic range may be achieved at the expense of a more complex layout.
In each of these schemes, each pixel has one light sensing element and the pixels are usually arranged in a square pattern, with same increment in both horizontal and vertical direction.
Another possible scheme is illustrated by U.S. Pat. No. 7,019,274, which shows a scheme in which a small light sensing element and a large light sensing element are incorporated in each pixel. By adjusting the light shield on top of the pixel to allow more exposure to the larger element and less exposure to the small element, the combined picture can have higher dynamic range. For such approach, however, the dynamic range extension is fixed once an imager is fabricated because the dynamic range extension depends only on the ratio of small and large elements and their aperture ratio. In practice, however, for different scenes, different dynamic ranges are needed. For example, in low light situations, it may be preferable to not use any high dynamic range image information, and instead, it may be desirable to expose each pixel to as much light as possible in order to achieve higher signal to noise ratio (SNR). However, the scheme used in U.S. Pat. No. 7,019,274 may not be optimized for such different schemes.